1. Field of the Invention
The present invention relates to fault-tolerant systems including multiplexed subsystems, and fault-tolerant control methods applied to fault-tolerant systems.
2. Description of the Related Art
Fault-tolerant systems (simply referred to as FT systems) have been developed as highly-reliable computer systems. For instance, Patent Document 1 discloses a fault-tolerant computer system, adopting an existing operating system (OS) precluding a duplex-redundancy functionality, which provides duplex fault-tolerant configurations each including a CPU, a VGA, and an I/O device. Upon occurrence of a fault, a routing controller rewrites an address of a request issued by the CPU so as to conceal duplex-redundancy functionality from the CPU.
Fault-tolerant systems are generally configured of multiplexed hardware modules, all of which are synchronized with each other in their operations. When a fault occurs in a certain part, a fault-tolerant system isolates a failed module while continuing processing with normal modules.
A fault-tolerant system is basically configured of duplex hardware modules, each including a CPU, a memory, and an I/O device, and a fault-tolerant controller (or an FT controller), coupled with hardware modules, which performs fault-tolerant control such as synchronous operation control and switching-at-failure control.
FIG. 9 is a block diagram showing the constitution of a fault-tolerant system 1001. The fault-tolerant system 1001 provides duplex CPU subsystems having the same configuration, namely a CPU subsystem 1100 including a main memory 1110, a CPU 1120, and an I/O hub 1130, and another CPU subsystem 1600 including a main memory 1610, a CPU 1620, and an IP hub 1630. Additionally, the fault-tolerant system 1001 provides duplex I/O subsystems, namely an I/O subsystem 1300 including I/O devices 1310 to 1330, and another I/O subsystem 1800 including I/O devices 1810 to 1830.
A fault-tolerant (FT) controller 1200 is interposed between the CPU subsystem 1100 and the I/O subsystem 1300 so as to control I/O operations between them. Another fault-tolerant (FT) controller 1700 is interposed between the CPU subsystem 1600 and the I/O subsystem 1800 so as to control I/O operations between them. The FT controllers 1200 and 1700 are connected together so as to maintain synchronous operations between subsystems, detect faults, and control isolation of failed modules.
It is possible to provide various methods for controlling isolation of failed modules, wherein FT systems are generally designed such that each module includes a hardware-isolation control section and a software-isolation control section.
For instance, the CPU subsystem 1100 including the main memory 1110 and the CPU 1120 is subject to hardware-isolation control because the CPU subsystem 1100 serves as a platform for executing software. Upon occurrence of an error in the CPU subsystem 1100, the FT controller 1200 (i.e. hardware) isolates the error-involved CPU subsystem 1100 from the FT system 1001 without affecting the other CPU subsystem 1600 which operates normally.
The two CPU subsystems 1100 and 1600 perform the same operation (hereinafter, referred to as a “lock step”) while being synchronized with each other on the clock basis. When one CPU subsystem fails, the FT controllers 1200 and 1700 logically isolate the failed CPU subsystem from the FT system 1001, so that the FT system 1001 maintains its operation by use of the other CPU subsystem.
When an IO device fails, it is possible to switch over the failed I/O device in accordance with software. When the I/O device 1310 fails, for example, the FT controller 1200 detects such a fault so as to send an error notice to the software (hereinafter, referred to as an “I/O device driver”) controlling the I/O device 1310. Subsequently, the I/O device driver stops using the failed I/O device 1310 while utilizing another I/O device 1810, which is a counterpart of the failed I/O device 1310 in a duplex-redundancy configuration.
The two I/O subsystems 1300 and 1800 are classified into an active I/O subsystem and a standby I/O subsystem. In a normal operation mode, both of the I/O subsystems 1300 and 1800 are accessible. Upon occurrence of a fault, the FT controllers 1200 and 1700 isolate the filed I/O subsystem, which is switched with another I/O subsystem. This switching process is called a failover.
As described above, both the CPU subsystems 1100 and 1600 are involved in the same operation (i.e. the lock step) whilst both the I/O subsystems 1300 and 1800 are accessible. Upon receiving an I/O request (an I/O transaction) issued by the CPU subsystem 1100 or 1600 in one module, the FT controller 1200 or 1700 conducts routing with one of the I/O subsystems 1200 and 1800. An access to an I/O subsystem of the other module is made via a cross-link L1010 interconnecting between the FT controllers 1200 and 1700.
The FT controllers 1200 and 1700 include voters 1220 and 1720 which compare two I/O requests output from the CPU subsystem 1100 and 1600. The voters 1220 and 1720 monitor existence/nonexistence of an error by successively comparing I/O requests output from the CPU subsystems 1100 and 1600. When two I/O requests match with each other, it is considered that no error occurs; hence, each voter selectively outputs a single I/O request to a target I/O device.
FIG. 10 illustrates an exemplary operation in which the voter 1220 selectively outputs a single I/O request. In this illustration, the CPUs 1120 and 1620 perform the same lock step so as to simultaneously output their I/O requests. The I/O request of the CPU 1120 is forwarded to the router 1210 via the I/O hub 1130. Similarly, the I/O request of the CPU 1620 is forwarded to the router 1710 via the I/O hub 1630. In this situation, the module 1010 serves as an active module whilst the module 1060 serves as a standby module, wherein the routers 1210 and 1710 involved in the lock step simultaneously forward the I/O requests, having the same content, to the voters 1220.
Upon receiving the I/O requests from the routers 1210 and 1710, the voter 1220 selectively outputs a single I/O request to the “target” I/O device 1320. Subsequently, the I/O device 1320 sends back a response to the I/O request, which is divided into two pieces by the FT controller 1200. Two pieces of response are reversely sent back to the CPU subsystems 1120 and 1620.
Compared to generally-used computer systems, fault-tolerant systems are involved in some restrictions in terms of hardware configurations. In particular, fault-tolerant systems are limited in the number of supportable I/O devices.
Generally speaking, FT systems are each configured of a single LSI chip, namely an FT control LSI chip, so that all I/O requests output from CPU subsystems are transmitted via FT controllers. For this reason, the FT control LSI suffers from a bottleneck of I/O performance in response to the increasing number of I/O devices connected to each FT system. To improve I/O performance of each FT system, it is necessary to connect I/O paths, such as plenty of PCI-Express buses (i.e. serial interfaces standardized by PCI-SIG, wherein the following description refers to adoption of x16 PCI-Express having sixteen channels), to FT controllers, and a cross-link having a large bandwidth (e.g. a communication path) needs to be connected between modules. This needs numerous high-speed interfaces in each FT control LSI chip, thus enlarging the sale of integrated circuitry and increasing cost.
FIG. 11 shows a modified configuration of the FT system 1001 in which each module is equipped with a single FT control LSI chip. Specifically, FT control LSI chips 1200 and 1700, each constituting a single FT controller, are each equipped with three PCI-Express buses. In this case, sixteen channels are provided per each PCI-Express bus; hence, each FT control LSI needs high-speed interfaces with 16×3=48 channels in total. Generally speaking, each FT control LSI is connected with numerous I/O paths requiring numerous high-speed interfaces, which may increase the size of an LSI chip. LSI chips with large sizes suffer from a low yield in manufacturing and a very high manufacturing cost.